Connectome-based machine learning models are vulnerable to subtle data manipulations

•Enhancement attacks falsely improve the performance of connectome-based models•Adversarial degrade models•Subtle data manipulations lead to large changes in In recent years, machine learning models using brain functional connectivity have furthered our knowledge brain-behavior relationships. The trustworthiness these has not yet been explored, and determining extent which can be manipulated change results is a crucial step understanding their trustworthiness. Here, we showed that only minor could drastically different performance. Although this work focuses on data, concepts investigated here apply any scientific research uses learning, especially with high-dimensional data. As becomes increasingly popular many fields research, may become major obstacle integrity learning. Neuroimaging-based predictive continue performance, widely overlooked aspect “trustworthiness,” or robustness manipulations. High imperative for researchers confidence findings interpretations. work, used connectomes explore how influence predictions. These included method enhance prediction adversarial noise designed changed model original were extremely similar (r = 0.99) did affect other downstream analysis. Essentially, connectome inconspicuously modified achieve desired Overall, enhancement evaluation existing highlight need counter-measures preserve academic potential translational applications. Human neuroimaging studies approaches identify associations generalize novel samples.1Whelan R. Garavan H. When optimism hurts: inflated predictions psychiatric neuroimaging.Biol. Psychiatry. 2014; 75: 746-748https://doi.org/10.1016/j.biopsych.2013.05.014Abstract Full Text PDF PubMed Scopus (129) Google Scholar,2Gabrieli J.D.E. Ghosh S.S. Whitfield-Gabrieli S. Prediction as humanitarian pragmatic contribution from human cognitive neuroscience.Neuron. 2015; 85: 11-26https://doi.org/10.1016/j.neuron.2014.10.047Abstract (353) Scholar They do so by aggregating weak informative signals occurring throughout brain.3Cremers H.R. Wager T.D. Yarkoni T. relation between statistical power inference fMRI.PLoS One. 2017; 12: e0184923https://doi.org/10.1371/journal.pone.0184923Crossref (204) Scholar,4Noble Mejia A.F. Zalesky A. Scheinost D. Improving magnetic resonance imaging moving beyond cluster-level inference.Proc. Natl. Acad. Sci. USA. 2022; 119 (e2203020119)https://doi.org/10.1073/pnas.2203020119Crossref (5) Machine (“connectome-based models”)5Shen X. Finn E.S. Rosenberg M.D. Chun M.M. Papademetris Constable R.T. Using modeling predict individual behavior connectivity.Nat. Protoc. 506-518https://doi.org/10.1038/nprot.2016.178Crossref (430) Scholar,6Cui Z. Gong G. effect regression algorithms sample size individualized behavioral features.Neuroimage. 2018; 178: 622-637https://doi.org/10.1016/j.neuroimage.2018.06.001Crossref (159) Scholar,7Du Y. Fu Calhoun V.D. Classification disorders connectivity: promising but challenging.Front. Neurosci. 525https://doi.org/10.3389/fnins.2018.00525Crossref are among most methods establishing relationships, they successfully characterized neural correlates various clinically relevant processes,8Rosenberg Casey B.J. Holmes A.J. complements explanation developing brain.Nat. Commun. 9: 589https://doi.org/10.1038/s41467-018-02887-9Crossref (90) including general ability,9Song Neural signatures attentional engagement during narratives its consequences event memory.Proc. 2021; 118 (e2021905118)https://doi.org/10.1073/pnas.2021905118Crossref (12) disorders,7Du Scholar,10Nielsen A.N. Barch D.M. Petersen S.E. Schlaggar B.L. Greene D.J. neuroimaging: evaluating applications psychiatry.Biol. Cogn. Neuroimaging. 2020; 5: 791-798https://doi.org/10.1016/j.bpsc.2019.11.007Abstract (42) affective states,11Goldfarb E.V. Seo Sinha Hippocampal seed predicts feeling stress.Nat. 11: 2650https://doi.org/10.1038/s41467-020-16492-2Crossref (27) abstinence individuals substance use disorder.12Yip S.W. Potenza M.N. Carroll K.M. Connectome-based cocaine abstinence.Am. J. 2019; 176: 156-164https://doi.org/10.1176/appi.ajp.2018.17101147Crossref (75) Recent uncovered bias, lack fairness across groups, models,13Benkarim O. Paquola C. Park B.-Y. Kebets V. Hong S.-J. de Wael R.V. Zhang Thomas Yeo B.T. Eickenberg M. Ge et al.The cost untracked diversity brain-imaging prediction.bioRxiv. (Preprint at)https://doi.org/10.1101/2021.06.16.448764Crossref Scholar,14Li Bzdok Chen Tam Ooi L.Q.R. Patil K.R. Jabbi Eickhoff S.B. al.Cross-ethnicity/race generalization failure resting-state connectivity.Sci. Adv. 8: eabj1812https://doi.org/10.1126/sciadv.abj1812Crossref (18) Scholar,15Greene A.S. Shen Noble Horien Hahn C.A. Arora Tokoglu F. Spann Carrión C.I. Barron D.S. al.Brain–phenotype fail who defy stereotypes.Nature. 609: 109-118https://doi.org/10.1038/s41586-022-05118-wCrossref (15) stereotypes.15Greene improvements accuracy6Cui (i.e., race, age, gender bias)13Benkarim improving quality clinical translation, accurate bias-free enough. should also high trustworthiness, define words, output trustworthy remains despite input X data). Without degree able ensuing interpretations, even modifications dramatically alter results. explored perspectives literature, privacy16Brundage Avin Wang Belfield Krueger Hadfield Khlaaf Yang Toner Fong al.Toward AI development: mechanisms supporting verifiable claims.arXiv. at)https://doi.org/10.48550/arXiv.2004.07213Crossref explainability,17Rawal Rawat Amant advances explainable artificial intelligence: status, challenges perspectives.IEEE Transactions Artificial Intelligence. 1: 1https://doi.org/10.1109/TAI.2021.3133846Crossref examine through lens manipulations.18Eshete B. Making trustworthy.Science. 373: 743-744https://doi.org/10.1126/science.abi5052Crossref (7) A form manipulation specific attacks), where pattern (or “noise”) deliberately trick added cause misclassification.19Szegedy Zaremba W. Sutskever I. Bruna Erhan Goodfellow Fergus Intriguing properties networks.arXiv. 2013; at)https://doi.org/10.48550/arXiv.1312.6199Crossref Scholar,20Goodfellow I.J. Shlens Szegedy Explaining harnessing examples.arXiv. at)https://doi.org/10.48550/arXiv.1412.6572Crossref contexts, cybersecurity,21Biggio Corona Maiorca Nelson Šrndić N. Laskov P. Giacinto Roli Evasion against at test time.in: Learning Knowledge Discovery Databases. Springer Berlin Heidelberg, 2013: 387-402https://doi.org/10.1007/978-3-642-40994-3_25Google Scholar,22Demontis Melis Biggio Arp Rieck K. Yes, more secure! case study android malware detection.IEEE Trans. Dependable Secure Comput. 16: 711-724https://doi.org/10.1109/TDSC.2017.2700270Crossref (108) image recognition,20Goodfellow Scholar,23Hendrycks Zhao Basart Steinhardt Song Natural examples.in: Proceedings IEEE/CVF Conference Computer Vision Pattern Recognition. 2021: 15262-15271https://doi.org/10.1109/CVPR46437.2021.01501Google medical recordings.24Paschali Conjeti Navarro Navab Generalizability vs. Robustness: investigating networks Medical Image Computing Assisted Intervention. 2018: 493-501https://doi.org/10.1007/978-3-030-00928-1_56Crossref (41) Scholar,25Finlayson S.G. Bowers J.D. Ito Zittrain J.L. Beam A.L. Kohane I.S. Adversarial learning.Science. 363: 1287-1289https://doi.org/10.1126/science.aaw4399Crossref (367) Scholar,26Han Hu Foschini L. Chinitz Jankelson Ranganath Deep electrocardiograms susceptible attack.Nat. Med. 26: 360-363https://doi.org/10.1038/s41591-020-0791-xCrossref (58) For neuroimaging, problematic distant future (e.g., applications25Finlayson Scholar,27Finlayson Chung H.W. deep Systems.arXiv. at)https://doi.org/10.48550/arXiv.1804.05296Crossref Scholar). immediate concern studies. majority seek perform ethical common than one might expect.28Acuna D.E. Brookes P.S. Kording K.P. Bioscience-scale automated detection figure element reuse.bioRxiv. at)https://doi.org/10.1101/269415Crossref (0) Scholar,29Bucci E.M. Automatic biomedical literature.Cell Death Dis. 400https://doi.org/10.1038/s41419-018-0430-3Crossref (19) Scholar,30Cicconet Elliott Richmond D.L. Wainstock Walsh Forensics: detecting duplication images manipulation-invariant similarity.arXiv. at)https://doi.org/10.48550/arXiv.1802.06515Crossref Scholar,31Bik Casadevall Fang F.C. prevalence inappropriate publications.mBio. 2016; 7: e00809-16https://doi.org/10.1128/mBio.00809-16Crossref (112) Scholar,32Fanelli How scientists fabricate falsify research? systematic review meta-analysis survey data.PLoS 2009; 4: e5738https://doi.org/10.1371/journal.pone.0005738Crossref (1007) Scholar,33Al-Marzouki Evans Marshall Roberts Are real? Statistical fabrication trials.BMJ. 2005; 331: 267-270https://doi.org/10.1136/bmj.331.7511.267Crossref (89) example, an analysis Bik al. about 2% biology papers contained evidence intentional manipulation.31Bik Furthermore, admitted fabrication/falsification, 14% seeing colleagues fabricate/falsify survey.32Fanelli result wasted grant money misdirection endeavors, enhanced diminished via crucial. models. Specifically, introduce “performance attack” models, injected small, inconspicuous patterns phenotype. We effectiveness Whereas manipulate particular prediction, modify entire dataset training data) both cases—enhancement attacks—we find subtle four datasets. demonstrate current implementations highly manipulations, points toward preventive measures built designs sharing practices. Functional MRI obtained Adolescent Brain Cognitive Development (ABCD) study,34Casey Cannonier Conley M.I. Cohen A.O. Heitzeg Soules M.E. Teslovich Dellarco D.V. adolescent development study: acquisition 21 sites.Dev. 32: 43-54https://doi.org/10.1016/j.dcn.2018.03.001Crossref (620) Connectome Project (HCP),35Van Essen D.C. Smith S.M. Behrens T.E.J. Yacoub E. Ugurbil WU-Minn HCP ConsortiumMinn ConsortiumThe Wu-Minn Project: overview.Neuroimage. 80: 62-79https://doi.org/10.1016/j.neuroimage.2013.05.041Crossref (2789) Philadelphia Neurodevelopmental Cohort (PNC),36Satterthwaite Connolly J.J. Ruparel Calkins Jackson M.A. Roalf D.R. Hopson Prabhakaran Behr Cohort: publicly available resource normal abnormal youth.Neuroimage. 124: 1115-1119https://doi.org/10.1016/j.neuroimage.2015.03.056Crossref (171) Southwest University Longitudinal Imaging Multimodal (SLIM) study.37Liu Wei Q. Meng Wu Bi Zuo X.-N. Qiu test-retest healthy young adults southwest China.Sci. Data. 170017https://doi.org/10.1038/sdata.2017.17Crossref (74) first three datasets (ABCD, HCP, PNC) IQ self-reported sex. SLIM was introduced measure (state anxiety). All analyses conducted SLIM, downloaded fully preprocessed connectomes. ABCD PNC, raw registered space previously described.38Greene Gao Task-induced state improves traits.Nat. 2807https://doi.org/10.1038/s41467-018-04920-3Crossref (202) Scholar,39Rapuano Maza M.T. Dennis N.J. Dorji Todd Behavioral vulnerability childhood.Dev. 46: 100878https://doi.org/10.1016/j.dcn.2020.100878Crossref (10) started minimally data.40Glasser M.F. Sotiropoulos S.N. Wilson J.A. Coalson T.S. Fischl Andersson Xu Jbabdi Webster Polimeni J.R. minimal preprocessing pipelines Project.Neuroimage. 105-124https://doi.org/10.1016/j.neuroimage.2013.04.127Crossref (2461) Next, standard, identical steps performed all BioImage Suite41Joshi Okuda Belhachemi Murphy Staib L.H. Unified framework development, deployment robust testing algorithms.Neuroinformatics. 2011; 69-84https://doi.org/10.1007/s12021-010-9092-8Crossref (133) (see experimental procedures). cases, parcellated into 268 nodes atlas.42Shen Groupwise whole-brain parcellation fMRI network node identification.Neuroimage. 82: 403-415https://doi.org/10.1016/j.neuroimage.2013.05.081Crossref (540) After excluding participants excessive motion (>0.2 mm), missing due full coverage, task 3,362 dataset, 506 562 PNC 445 remained. following sections, comprehensively characterize effects attacks, then evaluate attacks. show particularly vulnerable time, (Figure 1). To trust, trained baseline fluid intelligence (IQ) classification provide good benchmark because wide availability prominence literature.5Shen Scholar,43Biswal B.B. Mennes Gohel Kelly Beckmann C.F. Adelstein J.S. Buckner R.L. Colcombe discovery science function.Proc. 2010; 107: 4734-4739https://doi.org/10.1073/pnas.0911855107Crossref (2187) Scholar,44Gao Combining multiple phenotypic measures.Neuroimage. 201: 116038https://doi.org/10.1016/j.neuroimage.2019.116038Crossref (62) Scholar,45Weis Hoffstaedter Nostro B.T.T. Sex resting connectivity.Cereb. Cortex. 30: 824-835https://doi.org/10.1093/cercor/bhz129Crossref (69) Scholar,46Eliot Ahmed Khan Patel Dump “dimorphism”: comprehensive synthesis reveals few male-female differences size.Neurosci. Biobehav. Rev. 125: 667-697https://doi.org/10.1016/j.neubiorev.2021.02.026Crossref (84) ridge (rCPM)44Gao nested 10-fold cross-validation 10% feature selection. Regression evaluated Pearson’s correlation coefficient r cross-validated R2, called q2,47Scheinost Lake Salehi O’Connor al.Ten simple rules neuroimaging.Neuroimage. 193: 35-45https://doi.org/10.1016/j.neuroimage.2019.02.057Crossref measured predicted scores. found near zero correlations ABCD, consistent Li al.,14Li low Table S1). sex, linear support vector (SVM) logistic features L2 regularization. accuracy classifiers sex relatively success SVM regression, although had higher (Table will describe metrics S1 altered date, focused corrupting decrease accuracy.48Biggio Poisoning machines.arXiv. 2012; at)https://doi.org/10.48550/arXiv.1206.6389Crossref However, here, feasibility increase (which label “enhancement attacks”) ways cannot readily detected eye analyses. Current open standards would offer protection without statistically altering First, (n 3,262), (first session, n 506), 562) rCPM,44Gao selection correlated 2, top). randomly selected 20% edges same exact participants) individual-specific anti-correlated) each participant’s score. varied magnitude repeated cross-validation, recording 2). Results easily low-magnitude pattern, achieving near-perfect scores > 0.9) corrupted still maintained edge-wise ≈ counterparts. Figures 1 regularization parameters smaller parameter made effective addition, S2). suggest matrix undermine context appeared almost visually 2), if affected If common, affected, make it difficult determine manipulated. mean absolute value rCPM PNC. increased scale greatly increased, 0.9 (Figures 2 3A ). Corroborating visual inspection, remained very values 3A, top row). participant-wise Kolmogorov-Smirnov test49Massey F.J. goodness fit.J. Am. Stat. Assoc. 1951; 68-78https://doi.org/10.1080/01621459.1951.10500769Crossref (4071) suggested no significant edge distributions enhan